1. Box-Jenkins Differencing Method

This is called ARIMA modeling.



a) Backward and Difference Operator

  • Recall the backward operator \(B\), \[ B \, X_t = X_{t-1}. \]

  • Then define difference operator \(\bigtriangledown\) (del), \[ \bigtriangledown = (1-B). \]

  • For example, \[ \bigtriangledown X_t = (1-B)X_t = X_t - X_{t-1} \\ \\ \bigtriangledown^2 X_t = (1-B)(1-B) X_t = X_t - 2X_{t-1} + X_{t-2} \]



b) Differencing Linear Trend

  • Suppose your time series have a line plus a zero-mean staionary noise.
    \[ Y_t = a + bt + X_t \] That means \[ Y_{t-1} = a + b(t-1) + X_{t-1} \\ \\ \bigtriangledown Y_t = Y_t - Y_{t-1} = b + X_t - X_{t-1} \]

  • If \(X_t\) is stationary, then \(X_t-X_{t-1}\) is also stationary. Thus we have now \[ \bigtriangledown Y_t = \mu + M_t \] We can try to model this with ARMA(p,q) with intercept \(\mu = b\).

## [1] -9.461098



c) Differencing Quadratic Trend

  • If \(m_t\) is quadratic, then \[\begin{align} Y_t &= a + bt + c t^2 + X_t \\\\ Y_{t-1} &= a + b(t-1) + c (t-1)^2+ X_{t-1} \\\\ Y_{t-2} &= a + b(t-2) + c (t-2)^2 + X_{t-2}. \end{align}\]

    That means

    \[\begin{align} \bigtriangledown^2 Y_t &= Y_t - 2Y_{t-1} + Y_{t-2} \\\\ &= 2c + X_t - 2X_{t-1} + X_{t-2} \end{align}\]

    If \(X_t\) is stationary, so is \(X_t - 2X_{t-1} + X_{t-2}\).

  • If \(m_t\) is polynomial of deg \(k\), with coefficients $c_0, c_1, $ then applying \(k\) power of difference operator will remove the trend. \[ \bigtriangledown^k Y_t = k! c_k + \bigtriangledown^k X_t. \]

  • Then you will end up with some statonary series \(\bigtriangledown^k X_t\) with constant trend \(k! c_k\).

If you difference linar trend \(a+bt\) once, it becomes constant trend \(b\).

If you difference quadratic trend \(a+bt+ct^2\) twice, it becomes constant trend \(2c\).





3. Back to Modeling Lake HURON 4 different ways



a) Direct ARMA fit

Do not difference, force \(d=0\), and look for best ARMA(\(p,q\)).

## Series: Lake 
## ARIMA(1,0,1) with non-zero mean 
## 
## Coefficients:
##          ar1     ma1    mean
##       0.7665  0.3393  9.1290
## s.e.  0.0773  0.1123  0.3861
## 
## sigma^2 estimated as 0.4784:  log likelihood=-101.09
## AIC=210.18   AICc=210.62   BIC=220.48

##   B-L test H0: the sereis is uncorrelated
##   M-L test H0: the square of the sereis is uncorrelated
##   J-B test H0: the sereis came from Normal distribution
##   SD         : Standard Deviation of the series
##       BL15  BL20  BL25  ML15  ML20   JB    SD
## [1,] 0.963 0.952 0.934 0.567 0.641 0.89 0.684
## Series: Lake 
## ARIMA(1,0,1) with non-zero mean 
## 
## Coefficients:
##          ar1     ma1    mean
##       0.7665  0.3393  9.1290
## s.e.  0.0773  0.1123  0.3861
## 
## sigma^2 estimated as 0.4784:  log likelihood=-101.09
## AIC=210.18   AICc=210.62   BIC=220.48

Analysis 1 (direct fit)

  • auto.arima() chooses AR(2) with min AICc.

  • AR(2) with constant mean was fit directly to data. With observation \(Y_t\),

\[{\large Y_t \hspace{3mm} = \hspace{3mm} \mu + X_t }\] \[{\large \hspace{20mm} X_t \hspace{3mm} = \hspace{3mm} \phi_1 X_{t-1} + e_t + \theta_1 e_{t-1} }\] \[{\large e_t \sim WN(0, \sigma^2) }\]








b) ARMA with Linear Trend

  • If we choose that the level is going down linearly, we can try to fit a line with ARMA errors.

  • Force \(d=0\) (don’t difference)

  • Force linear regression by xreg=time(Lake)

## Series: Lake 
## Regression with ARIMA(1,0,1) errors 
## 
## Coefficients:
##          ar1     ma1  intercept     xreg
##       0.6682  0.3817    55.5443  -0.0242
## s.e.  0.0936  0.1136    18.0324   0.0094
## 
## sigma^2 estimated as 0.4612:  log likelihood=-98.66
## AIC=207.33   AICc=207.98   BIC=220.2

##   B-L test H0: the sereis is uncorrelated
##   M-L test H0: the square of the sereis is uncorrelated
##   J-B test H0: the sereis came from Normal distribution
##   SD         : Standard Deviation of the series
##       BL15  BL20  BL25  ML15  ML20    JB    SD
## [1,] 0.985 0.974 0.969 0.227 0.217 0.783 0.668
## Series: Lake 
## Regression with ARIMA(1,0,1) errors 
## 
## Coefficients:
##          ar1     ma1  intercept     xreg
##       0.6682  0.3817    55.5443  -0.0242
## s.e.  0.0936  0.1136    18.0324   0.0094
## 
## sigma^2 estimated as 0.4612:  log likelihood=-98.66
## AIC=207.33   AICc=207.98   BIC=220.2

Analysis 2 (linear trend)

  • Linear trend was fit to \(Y_t\), then the residuals were fitted with ARMA(1,1).

  • We observe $ Y_t = a+b t + X_t $ \(X_t\) is ARMA

  • In other words,

\[{\large Y_t \hspace{3mm} = \hspace{3mm} a + bt + X_t }\]

\[{\large \hspace{20mm} X_t \hspace{3mm} = \hspace{3mm} \phi_1 X_{t-1} + e_t + \theta_1 e_{t-1} }\]

\[{\large e_t \sim WN(0, \sigma^2) }\]








c) Arima(p,1,q)

  • Now difference once by setting \(d=1\).
## Series: Lake 
## ARIMA(1,1,2) 
## 
## Coefficients:
##          ar1      ma1      ma2
##       0.6385  -0.5349  -0.3514
## s.e.  0.1345   0.1445   0.1055
## 
## sigma^2 estimated as 0.4812:  log likelihood=-99.88
## AIC=207.76   AICc=208.2   BIC=218.02

##   B-L test H0: the sereis is uncorrelated
##   M-L test H0: the square of the sereis is uncorrelated
##   J-B test H0: the sereis came from Normal distribution
##   SD         : Standard Deviation of the series
##       BL15  BL20  BL25  ML15  ML20    JB    SD
## [1,] 0.979 0.956 0.942 0.524 0.495 0.577 0.677
## Series: Lake 
## ARIMA(1,1,2) 
## 
## Coefficients:
##          ar1      ma1      ma2
##       0.6385  -0.5349  -0.3514
## s.e.  0.1345   0.1445   0.1055
## 
## sigma^2 estimated as 0.4812:  log likelihood=-99.88
## AIC=207.76   AICc=208.2   BIC=218.02
## Series: Lake 
## ARIMA(1,1,2) with drift 
## 
## Coefficients:
##          ar1      ma1      ma2    drift
##       0.6911  -0.6271  -0.3729  -0.0241
## s.e.  0.0956   0.1251   0.1157   0.0100
## 
## sigma^2 estimated as 0.4663:  log likelihood=-99
## AIC=208.01   AICc=208.67   BIC=220.83

##   B-L test H0: the sereis is uncorrelated
##   M-L test H0: the square of the sereis is uncorrelated
##   J-B test H0: the sereis came from Normal distribution
##   SD         : Standard Deviation of the series
##       BL15  BL20  BL25  ML15  ML20    JB    SD
## [1,] 0.989 0.979 0.973 0.276 0.239 0.465 0.666

auto.arima() INCLUDES drift term by default.

Arima() DOES NOT include drift term by default when d=1.

To force drift term in Arima(), use include.drift=TRUE

Analysis 3 (take difference)

  • Difference of \(Y_t\) was taken. The difference seems to be ARMA(1,2).

  • In other words, $ Y_t $ is ARIMA(1,1,2), meaning \[{\large \bigtriangledown Y_t = Y_t - Y_{t-1} = X_t }\] \[{\large \hspace{20mm} X_t \hspace{3mm} = \hspace{3mm} \phi_1 X_{t-1} + e_t + \theta_1 e_{t-1} + \theta_2 e_{t-2} }\] \[{\large e_t \sim WN(0,\sigma^2) }\]








d) Arima(0,1,0)

  • difference once, and
## Series: Lake 
## ARIMA(0,1,0) 
## 
## sigma^2 estimated as 0.5383:  log likelihood=-106.49
## AIC=214.98   AICc=215.02   BIC=217.54

##   B-L test H0: the sereis is uncorrelated
##   M-L test H0: the square of the sereis is uncorrelated
##   J-B test H0: the sereis came from Normal distribution
##   SD         : Standard Deviation of the series
##       BL15  BL20  BL25  ML15  ML20    JB    SD
## [1,] 0.169 0.167 0.125 0.161 0.098 0.369 0.733

Analysis 4 (take difference)

  • Difference of \(Y_t\) was taken. The difference seems to be WN.

  • In other words, $ Y_t $ is ARIMA(0,1,0), meaning \[{\large \bigtriangledown Y_t = Y_t - Y_{t-1} = X_t }\] \[{\large \hspace{20mm} X_t \hspace{3mm} = \hspace{3mm} e_t }\] \[{\large e_t \sim WN(0,\sigma^2) }\]




Conclusions

a) ARIMA(1,0,1) with mean is fitting adequately. This suggests that Lake is stationary (at constant mean) without any transformation.

\[{\large Y_t \hspace{3mm} = \hspace{3mm} \mu + X_t, \hspace{10mm} X_t \mbox{ is ARMA(1,1)} }\]



b) ARIMA(1,0,1) with linear trend is fitting adequately. Intercept and slope are significant.

\[{\large Y_t \hspace{3mm} = \hspace{3mm} a + bt + X_t, \hspace{10mm} X_t \mbox{ is ARMA(1,1)} }\]



c) ARIMA(1,1,2) with drift is fitting adequately. Drift is significant, and same as the slope in (b).

\[{\large \bigtriangledown Y_t = Y_t - Y_{t-1} = b + X_t, \hspace{10mm} X_t \mbox{ is ARMA(1,2)} }\]



d) ARIMA(0,1,0) is fitting adequately. This suggests that Lake is RW w/o drift.

\[{\large \bigtriangledown Y_t = Y_t - Y_{t-1} = X_t = e_t \hspace{10mm} e_t \mbox{ is ARMA(0,0) = WN} }\]



(b) and (c) are concluding the same model.

(a) is a weaker version of (b), but intercept and slope in (b) are significant.

(d) is a weaker version of (c), but ARMA parameters in (c) are significant.

The model (b) (which is essentially same as (c)) is the best candidate.





Summary

If you difference linar trend \(a+bt\) once, it becomes constant trend \(b\).

If you difference quadratic trend \(a+bt+ct^2\) twice, it becomes constant trend \(2c\).

If you difference Random Walk without drift once, it becomes N(\(0,\sigma^2\)) iid noise.

If you difference Random Walk with drift \(\delta\) once, it becomes N(\(\delta,\sigma^2\)) iid noise.






TS Class Web PageR resource page